scrapy crawler for CVE

How to use it

Pre-Request

• Git
• Python & scrapy

Install and Run

└─$ git clone
└─$ cd scrapy_CVE; scrapy crawl spider_XXXX

After running the command, you can sit back and chill. It will take certain amount of time depend on the scope you crawl and network bandwith you have.

Result and Explain

This script aimed at spider all CVE in CVE-list directories. The script only collect 5 culums, which were CVE, CVSS score, Product Type, Vendor, and Product Name, which were shown below.

└─$ head 2020_Hardware.csv 
cve,severity,product_type,vendor,product
CVE-2021-0144,4.6,Hardware,Intel,Core I7-3820
CVE-2021-0144,4.6,Hardware,Intel,Core I7-3920xm
CVE-2021-0144,4.6,Hardware,Intel,Core I7-3930k
CVE-2021-0144,4.6,Hardware,Intel,Core I7-3940xm
CVE-2021-0144,4.6,Hardware,Intel,Core I7-3960x
CVE-2021-0144,4.6,Hardware,Intel,Core I7-3970x
CVE-2021-0144,4.6,Hardware,Intel,Core I7-4820k
CVE-2021-0144,4.6,Hardware,Intel,Core I7-4930k
CVE-2021-0158,4.6,Hardware,Intel,Celeron N2806
CVE-2021-0144,4.6,Hardware,Intel,Core I7-4930mx
CVE-2021-0158,4.6,Hardware,Intel,Celeron N2807
CVE-2021-0157,4.6,Hardware,Intel,Celeron N2806
CVE-2021-0144,4.6,Hardware,Intel,Core I7-4940mx

It's was a simple script wrote for my colleague.